Apparatus and method for processing three dimensional image on multi-layer display

ABSTRACT

An apparatus and method of processing three-dimensional (3D) images on a multi-layer display may generate virtual depth information based on original depth information, and display 3D images having various depth values using the generated virtual depth information. Also, the apparatus and method may appropriately provide color information to each of a plurality of display layers, thereby preventing an original image from being damaged.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of Korean Patent Application No.10-2008-117629, filed on Nov. 25, 2008, in the Korean IntellectualProperty Office, the disclosure of which is incorporated herein byreference.

BACKGROUND

1. Field

One or more example embodiments of the present disclosure relate to atechnique for displaying three-dimensional (3D) images on a multi-layerdisplay.

2. Description of the Related Art

Binocular parallax may exist due to eyes of a person spaced apart fromeach other by a predetermined distance. The term binocular parallaxdescribes a disparity between the two retinal images of athree-dimensional object or scene arising from the slightly differentvantage points of the two eyes. Binocular disparity functions as one ofthe binocular cues of visual depth perception and provides the basis forstereopsis. For example, an observer may have an illusion ofthree-dimensionality due to binocular parallax. Recently, techniques fordisplaying 3D images using a binocular parallax principle are gaininginterest in a variety of fields.

Techniques for displaying 3D images may be classified into a shutterglass method, an autostereoscopic method, a holography method, and thelike. A user may need to wear a separate device such as polarizationglasses and the like in the shutter glass method, and the user may view3D images only in a predetermined location through the autostereoscopicmethod. Accordingly, to overcome the above-mentioned problems in theshutter glass method and the autostereoscopic method, the holographymethod has recently been a topic of increased study.

As one holography method, a volumetric 3D display technique may beprovided. The volumetric 3D display technique may display 3D imagesusing an optical illusion that occurs when a user views images projectedon a plurality of display layers. However, the user may not have anenriched perception of depth, as necessary, despite using the pluralityof display layers.

SUMMARY

According to example embodiments, an apparatus of processing images fora multi-layer display may be provided. The apparatus may include a depthinformation conversion unit to convert original depth information of aninput image to generate virtual depth information, and a colorinformation conversion unit to adjust color information of the inputimage based on the virtual depth information to provide output colorinformation to each of a plurality of display layers.

In this instance, the depth information conversion unit may convert theoriginal depth information of the input image using histogramequalization to generate the virtual depth information.

Also, the color information conversion unit may adjust saturation andbrightness of the input image while maintaining hue of the input imagebased on the virtual depth information to thereby provide the outputcolor information to each of the plurality of display layers.

According to example embodiments, a method of processing images for amulti-layer display may be provided. The method may include convertingoriginal depth information of an input image to generate virtual depthinformation, and adjusting color information of the input image based onthe virtual depth information to provide output color information toeach of a plurality of display layers.

One or more example embodiments of the present disclosure may provide anapparatus and method of processing images for a multi-layer display thatmay use virtual depth information generated by converting original depthinformation, thereby providing three-dimensional (3D) images providingan enriched depth feeling, i.e. an enhanced perception of depth, to auser.

One or more example embodiments of the present disclosure may provide anapparatus and method of processing images for a multi-layer display thatmay generate virtual depth information utilizing histogram equalization,thereby enhancing a depth feeling.

One or more example embodiments of the present disclosure may provide anapparatus and method of processing images for a multi-layer display thatmay appropriately generate color information for a plurality of displaylayers, thereby providing 3D images having various depth feelings whilemaintaining hues of original images.

One or more example embodiments of the present disclosure may provide anapparatus and method of processing images for a multi-layer display thatmay utilize a local dimming controller, thereby maximizing a 3Dexperience.

Additional aspects and/or advantages will be set forth in part in thedescription which follows and, in part, will be apparent from thedescription, or may be learned by practice of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

These and/or other aspects and advantages will become apparent and morereadily appreciated from the following description of the embodiments,taken in conjunction with the accompanying drawings of which:

FIG. 1 illustrates an example of a two-dimensional RGB (Red, Green, andBlue) of an input image, an original depth map, and a virtual depth mapaccording to a related art.

FIG. 2 illustrates a comparison with respect to original depthinformation and virtual depth information according to exampleembodiments;

FIG. 3 is a block diagram illustrating an apparatus of processing imagesfor a multi-layer display according to example embodiments;

FIG. 4 is an operational flowchart illustrating operations of a colorinformation conversion unit according to example embodiments;

FIG. 5 is a cross-sectional diagram illustrating a HSV (Hue, Saturation,and Value) color space used for describing a back layer or a front layereach functioning as a window according to a related art

FIG. 6 is a graph illustrating operations of a local dimming controlleraccording to example embodiments;

FIG. 7 is a conceptual diagram illustrating three display layersaccording to example embodiments;

FIG. 8 is a conceptual diagram illustrating N display layers accordingto example embodiments; and

FIG. 9 is an operational flowchart illustrating a method of generatingcolor information for a K-th display layer and a (K+1)-th display layerin the presence of N display layers according to example embodiments.

DETAILED DESCRIPTION

Reference will now be made in detail to example embodiments, which areillustrated in the accompanying drawings, wherein like referencenumerals refer to the like elements throughout. Example embodiments aredescribed below to explain the present disclosure by referring to thefigures.

FIG. 1 illustrates an example of a two-dimensional (2D) RGB (Red, Green,and Blue) of an input image, an original depth map, and a virtual depthmap. FIG. 2 illustrates a histogram with respect to original depthinformation and virtual depth information.

Referring to FIG. 1, a 2D RGB 110 of the input image may be provided asan image processing apparatus. At a later time, the 2D RGB 110 of theinput image may be used in generating output color information for eachof a plurality of display layers.

A reference segment 120 designates an original depth map of the inputimage depending on original depth information of the input image. Areference segment 130 designates a virtual depth map depending on thevirtual depth information generated by converting the original depthinformation.

When comparing the original depth map 120 and the virtual depth map 130,a user may experience a more enriched depth feeling from the virtualdepth map 130 than experienced from the original depth map 120.

Referring to FIG. 2, it is assumed that the closer an object is locatedto an observer, the closer a depth value of the object, indicating adegree of the depth feeling, approaches ‘1’, and the further the objectis located away from the observer, the closer the depth value of theobject approaches ‘0’. Specifically, concerning a layer 1 and a layer 2,the further the object is located away from the observer, the closer theobject seems to be viewed to the layer 1, and the closer the object islocated to the observer, the closer the object seems to be viewed to thelayer 2.

Histogram 210 designates a histogram with respect to original depthinformation corresponding to the original depth map 120 illustrated inFIG. 1. In histogram 210, available depth information elements of depthinformation elements made up of depth information are densely populatedin a predetermined range. Specifically, the original depth informationmay have a brightness value (V) greater than ‘0’ only in a range A and arange B, and have a brightness value (V) of ‘0’ (nearly ‘0’) inremaining ranges other than the ranges A and B.

Histogram 220 designates a histogram with respect to virtual depthinformation corresponding to the virtual depth map 130 illustrated inFIG. 1. Here, histogram 220 may be generated by applying histogramequalization to histogram 210. In histogram 220, brightness valuesgreater than ‘0’ exist in a range of a depth value of 0 to 1, and thusthe available depth information elements may be dispersively present. Asa result, a user may experience a more enriched depth feeling.

Although described in detail below, according to an apparatus and methodof processing images according to example embodiments, the originaldepth information may be converted into the virtual depth information sothat the user may experience a more enriched depth feeling.

FIG. 3 is a block diagram illustrating an apparatus of processing imagesfor a multi-layer display according to example embodiments.

Referring to FIG. 3, the apparatus according to the present exampleembodiment includes, for example, a depth information conversion unit310 and a color information conversion unit 320.

The depth information conversion unit 310 converts original depthinformation (Depth_in) of an input image to generate virtual depthinformation (Depth′). Here, the virtual depth information (Depth′) maybe generated by various methods.

For example, the depth information conversion unit 310 may convert theoriginal depth information (Depth_in) of the input image using histogramequalization to generate the virtual depth information (Depth′). Also,the depth information conversion unit 310 may convert the original depthinformation (Depth_in) of the input image to generate the virtual depthinformation (Depth′), so that available depth information elements,which are densely populated in a predetermined range, included in theoriginal depth information (Depth_in) of the input image are dispersed.In addition, the depth information conversion unit 310 may enlarge adistance between the available depth information elements being denselypopulated in the predetermined range to thereby generate the virtualdepth information (Depth′).

Consequently, available depth information elements included in thevirtual depth information (Depth′) are widely present, and thus the usermay experience a more enriched depth feeling from the virtual depthinformation (Depth′) than from the original depth information(Depth_in).

Also, the color information conversion unit 320 adjusts colorinformation of the input image based on the virtual depth information(Depth′) to provide output color information to each of a plurality oflayers (layer 1, layer 2, . . . , and layer N). In this instance, thecolor information conversion unit 320 may adjust only a brightness andsaturation of the input image while maintaining hue of the input imageto thereby generate the output color information. In particular, thecolor information conversion unit 320 may generate output colorinformation so that a color of an image viewed by a user is notdifferent than a color of the input image.

In this instance, each of the plurality of layers (layer 1, layer 2, . .. , and layer N) may display output images on a display device using theoutput color information, and the user may view three-dimensional (3D)images on the display device through the displayed output images.

The color information converting unit 320 may include a color spaceconversion device 321, a medium color information generation unit 322,and a plurality of output color information generation units 323, 324,and 325.

When the color information is an RGB input (RGB_in), the color spaceconversion device 321 converts the RGB input into a HSV (Hue,Saturation, and Value) format. S and V values of the converted HSV maybe provided to the medium color information generation unit 322.

Also, the medium color information generation unit 322 may receive layerinformation of each of the plurality of layers (layer 1, layer 2, . . ., and layer N), the S and V values of the input image, and the virtualdepth information (Depth′). Also, the medium color informationgeneration unit 322 may generate medium color information for each ofthe plurality of layers (layer 1, layer 2, . . . , and layer N) usingthe layer information of each of the plurality of layers, the S and Vvalues of the input image, and the virtual depth information (Depth′).Here, the medium color information includes S and V values for each ofthe plurality of layers (layer 1, layer 2, . . . and layer N). In thisinstance, the medium color information may be generated to prevent acolor of a 3D image viewed by a user from being different than anoriginal color of the input image.

Also, the medium color information generation unit 322 may provide S1and V1 of medium color information for the layer 1, may provide S2 andV2 of medium color information for the layer 2, and may provide SN andVN of medium color information for the layer N, to a first output colorinformation generation unit 323, a second output color informationgeneration unit 324, and an N-th output color information generationunit 325, respectively.

In this instance, each of the output color information generation units323, 324, and 325 converts, into RGB format, an H value provided fromthe color space conversion device 321 and the S and V values providedfrom the medium color information generation unit 322. Also, each of theoutput color information generation units 323, 324, and 325 provides R1,G1, B1, R2, G2, B2, and RN, GN, BN to each of the plurality of layers(layer 1, layer 2, . . . , and layer N).

Each of the layers (layer 1, layer 2, . . . , and layer N) may displaythe output image using the provided RGB values, and a user may view 3Dimages from the output image of each of the layers.

Also, a local dimming controller 330 may control local dimming withrespect to an LED Back Light Unit (BLU) based on the virtual depthinformation so as to improve a 3D experience. Specifically, the localdimming controller 330 may control a power of the LED BLU so as toreduce power consumption and improve contrast. For example, a relativelystrong power may be applied to an object located relatively closer tothe observer, and a relatively weak power may be applied to an objectlocated relatively further away from the observer.

In this instance, since the number of LEDs is generally smaller than thenumber of pixels, the local dimming controller 330 may blur a depth mapof the virtual depth information, thereby obtaining an average of depthvalues with respect to a plurality of pixels corresponding to a singleLED. Also, the local dimming controller 330 may control a power appliedto the LED based on the average of the depth values with respect to theplurality of pixels corresponding to the single LED.

FIG. 4 is an operational flowchart illustrating operations of a colorinformation conversion unit according to example embodiments.

Here, it is assumed that a number of display layers is two, and adisplay layer located closer to the observer is denoted as a frontlayer, and a display layer located further away from the observer isdenoted as a back layer.

In a structure including two display layers, the back layer may displayan object located furthest away from the observer, and function as aback light with respect to the front layer. Also, the front layer maydisplay an object located closest to the observer, and function as awindow with respect to the back layer. Concerning the above-mentionedfunctioning of the back layer and front layer, an object displayed onthe back layer or front layer may need to be processed.

Referring to FIG. 4, in operation S410, the color information conversionunit may recognize virtual depth information (Depth′) and V and S of theinput image. Here, it is assumed that the color information conversionunit ascertains H of the input image in advance.

Also, in operation S420, the color information conversion unit maycalculate a brightness value (Back_V) and saturation value (Back_S) forthe back layer concerning the virtual depth information (Depth′) and arelation or function between the back layer and the front layer.

For example, the color information conversion unit may determine anoriginal brightness value and saturation value as the brightness value(Back_V) and saturation value (Back_S) for the back layer with respectto an object located further away from the observer and displayed on theback layer. In this instance, the color information conversion unitdetermines white as a color for the front layer. Here, determining thecolor for the front layer as white may indicate maximizing thebrightness value for the front layer and minimizing the saturation valuefor the front layer. In this instance, the front layer may function as awindow with respect to the back layer.

In another example, the color information conversion unit may determinewhite as a color for the back layer, with respect to an object locatedcloser to the observer and displayed on the front layer. In thisinstance, the color information conversion unit may determine anoriginal brightness value and saturation value as the brightness valueand saturation value for the front layer. Here, the back layer mayfunction as a back light with respect to the front layer.

Also, the color information conversion unit may calculate an appropriatebrightness value (Back_V) and saturation value (Back_S) for the backlayer with respect to an object, so as to allow a viewer to moderatelyexperience a depth feeling between the back layer and front layer,concerning a relation between the back layer and front layer.

Also, in operation S430, the color information conversion unitcalculates a brightness value (Front_V) and saturation value (Front_S)for the front layer concerning the brightness value (Back_V) andsaturation value (Back_S) for the back layer, and the virtual depthinformation (Depth′).

In this instance, the color information conversion unit may calculatethe brightness values (Front_V) and saturation value (Front_S) for thefront layer to prevent a color of a 3D image viewed by a user from beingdifferent than a color of the input image.

Consequently, the color information conversion unit may calculate thebrightness value (Back_V) and saturation value (Back_S) for the backlayer, and the brightness value (Front_V) and saturation value (Front_S)for the front layer using Equation 1 below, for example.

Back_(—) V=(1−V)*Depth′+V,

Back_(—) S=S*(1−Depth′),

Front_(—) V=V/(Back_(—) V), and

Front_(—) S=S−Back_(—) S.  Equation 1

FIG. 5 is a cross-sectional diagram illustrating an HSV (Hue,Saturation, and Value) color space used for describing a back layer or afront layer each functioning as a window according to exampleembodiments.

Referring to FIG. 5, in a cross-sectional area of the HSV color space, abrightness value (V) and a saturation value (S) may increase in adirection of an arrow, respectively. It is assumed that a brightnessvalue (V) and saturation value (S) of an input image corresponds to apointer A.

When an object is located furthest away from an observer, that is,having a depth value of the input image (or pixel) of ‘0’, an original Vand S of the input image may be assigned to the back layer. In thisinstance, a brightness value and saturation value for the front layermay be determined as ‘1’ and ‘0’, respectively, so that the front layerfunctions as a window.

Conversely, when an object is located closest to the observer, that is,having the depth value of the input image of ‘1’, the original V and Sof the input image may be assigned to the front layer. In this instance,the brightness value and saturation value for the back layer may bedetermined as ‘1’ and ‘0’, respectively, so that the back layerfunctions as a back light.

Also, when the depth value of the input image is in a range of ‘0’ to‘1’, the brightness value and saturation value for the back and frontlayers may be appropriately determined through Equation 1 above.

FIG. 6 is a graph illustrating operations of a local dimming controlleraccording to example embodiments.

It is assumed that an object is located further away from an observeralong with a reduction in virtual depth (Depth′)

Referring to FIG. 6, the local dimming controller controls a power ofthe LED BLU depending on the virtual depth to reduce power consumptionand improve a contrast.

Specifically, the local dimming controller may apply a relatively strongpower to an object located closer to the observer, and apply arelatively weak power to an object further away from the observer.

FIG. 7 is a conceptual diagram illustrating three display layersaccording to example embodiments, and FIG. 8 is a conceptual diagramillustrating N display layers according to example embodiments.

Referring to FIGS. 7 and 8, the apparatus and method of processingimages according to present example embodiments may be well applicableeven in a structure having three or more display layers.

Referring to FIG. 7, in the structure having the three display layers,an object having depth values belonging to a range 1 is displayedbetween a layer 1 and a layer 2, and an object having depth valuesbelonging to a range 2 is displayed between a layer 2 and a layer 3.

Also, referring to FIG. 8, an object having depth values belonging to arange N-1 is displayed between a layer N-1 and a layer N.

In FIG. 7, when the object having the depth values belonging to therange 1 is expressed, the layer 1 may function as a back layer and thelayer 2 may function as a front layer. In this instance, the layer 3 maybe processed as a window for the layer 1 and layer 2.

Similarly, in FIG. 8, when an object having depth values belonging to arange K is expressed, a layer K may function as the back layer, and alayer K+1 may function as the front layer. In this instance, the layer 1to the layer K−1 may function as the back light for the layer K, and alayer K+2 to the layer N may function as the window for the layer K andlayer K+1.

Consequently, descriptions of FIGS. 1 to 6 may be applicable even in thestructure having three or more display layers as illustrated in FIGS. 7and 8. This will be described in detail with reference to FIG. 9.

FIG. 9 is an operational flowchart illustrating a method of generatingcolor information for a K-th display layer and a (K+1)-th display layerin the presence of N display layers according to example embodiments.

Referring to FIG. 9, in operation S910, the color information conversionunit recognizes V, S, and virtual depth information (Depth′) of an inputimage.

Also, the color information conversion unit estimates, based on thevirtual depth information (Depth′), on which range an image is displayedfrom among N-1 ranges classified by N display layers. Also, in operationS920, the color information conversion unit selects a K-th layer and a(K+1)-th layer according to an estimated result. In this instance, theK-th layer may function as the back layer, and the (K+1)-th layer mayfunction as the front layer.

In operation S930, the color information conversion unit calculates abrightness value (Back_V) and saturation value (Back_S) for the K-thlayer functioning as the back layer based on the virtual depthinformation (Depth′).

In operation S940, the color information conversion unit calculates abrightness value (Front_V) and saturation value (Front_S) for the(K+1)-th layer functioning as the front layer based on the brightnessvalue (Back_V) and saturation value (Back_S) for the K-th layer.

In this instance, in operation S950, the color information conversionunit determines a brightness value for remaining layers as ‘1’ and asaturation value for the remaining layers as ‘0’, so that the remaininglayers may function as the window or back light.

The method of processing images according to the above-describedexemplary embodiments may be recorded as computer readablecode/instructions in/on a computer-readable media including programinstructions to implement various operations embodied by a computer. Themedia may also include, alone or in combination with the programinstructions, data files, data structures, and the like. Examples ofcomputer-readable media include magnetic media such as hard disks,floppy disks, and magnetic tape; optical media such as CD ROM disks andDVDs; magneto-optical media such as floptical disks; and hardwaredevices that are specially configured to store and perform programinstructions, such as read-only memory (ROM), random access memory(RAM), flash memory, and the like. Examples of program instructionsinclude both machine code, such as produced by a compiler, and filescontaining higher level code that may be executed by the computer usingan interpreter. The described hardware devices may be configured to actas one or more software modules in order to perform the operations ofthe above-described exemplary embodiments, or vice versa.

Although a few exemplary embodiments have been shown and described, itwould be appreciated by those skilled in the art that changes may bemade to these exemplary embodiments without departing from theprinciples and spirit of the present disclosure, the scope of which isdefined by the claims and their equivalents.

1. An apparatus of processing images for a multi-layer display, theapparatus comprising: a depth information conversion unit to convertoriginal depth information of an input image to generate virtual depthinformation; and a color information conversion unit to adjust colorinformation of the input image based on the virtual depth information toprovide output color information to each of a plurality of displaylayers.
 2. The apparatus of claim 1, wherein the depth informationconversion unit converts the original depth information of the inputimage using histogram equalization to generate the virtual depthinformation.
 3. The apparatus of claim 1, wherein the depth informationconversion unit converts the original depth information so thatavailable depth information elements included in the original depthinformation of the input image are dispersed, the available depthinformation elements being densely populated in a predetermined rangebefore being dispersed.
 4. The apparatus of claim 1, wherein the depthinformation conversion unit increases distances between available depthinformation elements included in the original depth information of theinput image to thereby generate the virtual depth information.
 5. Theapparatus of claim 1, wherein the color information conversion unitadjusts saturation and brightness of the input image while maintaininghue of the input image based on the virtual depth information to therebyprovide the output color information to each of the plurality of displaylayers.
 6. The apparatus of claim 1, wherein the color informationconversion unit provides the output color information to each of theplurality of display layers based on a relation of the front-rearposition between the plurality of display layers with respect to anobserver or backlight, and based on the virtual depth information. 7.The apparatus of claim 1, wherein the color information conversion unitadjusts brightness and saturation of the input image based on thevirtual depth information to generate the output color information forat least one display layer from among the plurality of display layers,and generates the output color information for at least one remainingdisplay layer based on the output color information for the at least onedisplay layer.
 8. The apparatus of claim 1, wherein the plurality ofdisplay layers display output images using output color information, andthe color information conversion unit generates the output colorinformation to prevent displayed images, generated due to the outputimages, from distorting the color information of the input image.
 9. Theapparatus of claim 1, further comprising: a local dimming controller tocontrol local dimming applied to the input image depending on thevirtual depth information.
 10. The apparatus of claim 1, wherein thecolor information conversion unit includes: a color space conversiondevice to convert a format of the color information of the input imagefrom an RGB (Red, Green, Blue) format to a HSV (Hue, Saturation, andValue) format.
 11. The apparatus of claim 10, wherein the colorinformation conversion unit further includes: a medium color informationgeneration unit to adjust the color information of the input image inthe HSV format using the virtual depth information to generate mediatecolor information for each of the plurality of display layers; and anoutput color information generation unit to convert a format of themediate color information into the RGB format to generate the outputcolor information, and to provide the output color information to theplurality of display layers.
 12. A method of processing images for amulti-layer display using an image processing apparatus, the methodcomprising: converting original depth information of an input image togenerate virtual depth information; and adjusting, by the imageprocessing apparatus, color information of the input image based on thevirtual depth information to provide output color information to each ofa plurality of display layers.
 13. The method of claim 12, wherein theconverting converts the original depth information of the input imageusing histogram equalization to generate the virtual depth information.14. The method of claim 12, wherein the converting converts the originaldepth information, so that available depth information elements includedin the original depth information of the input image are dispersed, theavailable depth information elements being densely populated in apredetermined range before being dispersed.
 15. The method of claim 12,wherein the adjusting adjusts saturation and brightness of the inputimage while maintaining hue of the input image based on the virtualdepth information to thereby provide the output color information toeach of the plurality of display layers.
 16. The method of claim 12,further comprising: controlling local dimming applied to the input imagedepending on the virtual depth information.
 17. At least one computerreadable storage medium storing computer readable code comprisinginstructions implementing the method of claim 12.